feat: add ssz to engine api#764
Conversation
…eflect optionality removal Removes `Optional[T]` mapping as it is replaced by specific zero/empty value encoding in container definitions. Updates `PayloadStatusV1`, `ForkchoiceUpdatedResponseV1`, `ExecutionPayloadBodyV1`, and `ExecutionPayloadBodyV2` to use non-optional types where JSON mapping implies a zero/empty value instead of true optionality. Updates SSZ mappings for `engine_getPayloadBodiesByHashV1/V2` and `engine_getBlobsV1/V2/V3` to use nested lists (`List[List[T, 1]]`) instead of `Optional[T]` to represent the presence or absence of data, consistent with non-null SSZ encoding for absent data. Adds notes explaining the zero/empty encoding for absent fields.
…update examples The documentation for how `T or null` maps to SSZ encoding is being clarified. Replaced the generic statement about `Optional[T]` being encoded as `List[T, 1]` with a more direct explanation that it is represented as `List[T, 1]`. Also updated the description for `payload_attributes` in `ForkchoiceUpdated` requests to explicitly state that presence is indicated by a list with 1 element, matching the SSZ type definition. Additionally, added missing vocabulary words to `wordlist.txt` to improve future documentation generation tools.
…ation to reflect capability exchange The transport negotiation mechanism has been updated to exclusively use `engine_exchangeCapabilities` over JSON-RPC for determining support of SSZ REST endpoints. This change clarifies the required steps for clients to discover and utilize the binary SSZ transport.
Update documentation to better explain how JSON-RPC remains the default for negotiation and fallback, explicitly stating when binary SSZ is used. This clarifies the steps involved for CL and EL during initialization.
| - The CL uses SSZ natively, forcing a round-trip conversion (SSZ to JSON, then JSON to internal types) at the Engine API boundary. | ||
|
|
||
| Binary SSZ eliminates all of this. The CL sends raw SSZ bytes over HTTP; the EL deserializes directly. No hex encoding, no JSON parsing, no intermediate representations. Payload sizes are reduced by 50% or more compared to JSON-RPC, and serialization is no longer a bottleneck in the critical path between CL and EL. |
There was a problem hiding this comment.
The EL uses RLP, why not RLP? The EL does not currently support SSZ while the CL does support RLP for various reasons. This would reduce the number of libraries in the EL but be net zero for the CL.
What unique utility does SSZ provide?
There was a problem hiding this comment.
The CL does not support RLP, some clients do, but it is unnecessary, it is an optimization.
|
|
||
| ### Binary SSZ transport | ||
|
|
||
| Clients **MAY** support a binary SSZ transport as an alternative to JSON-RPC. The binary transport uses resource-oriented REST endpoints with raw SSZ request and response bodies (`application/octet-stream`), eliminating JSON and hex-encoding overhead for fast CL-EL communication. Endpoints follow Beacon API conventions with path-based versioning (e.g., `POST /engine/v5/payloads`). |
There was a problem hiding this comment.
Current specs define the capabilities as JSON-RPC methods names like:
- engine_newPayloadV2
- engine_getPayloadV5
This suggestion change the capabilities vocabulary to strings like POST /engine/v5/payloads.
This is protocol change, not just a transport addition.
There was a problem hiding this comment.
why would it be a protocol change? We just using the v before not after.
There was a problem hiding this comment.
Explained in detail in this thread #764 (review)
|
|
||
| --- | ||
|
|
||
| #### `POST /engine/v1/capabilities` — Exchange capabilities |
There was a problem hiding this comment.
Related to earlier point, If we want to add this endpoint then we should remove engine_exchangeCapabilities reference from the documents and that will be a protocol change.
Or we keep both but must keep the return format of both to one canonical format, and that will existing response format of the engine_exchangeCapabilities to avoid a breaking change to protocol.
There was a problem hiding this comment.
I don't see how this is protocol breaking change?
There was a problem hiding this comment.
Explained in detail in this thread #764 (review)
| ### Client errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `400` | Bad Request | Malformed SSZ encoding | | ||
| | `401` | Unauthorized | Missing or invalid JWT token | | ||
| | `404` | Not Found | Unknown payload ID | | ||
| | `409` | Conflict | Invalid forkchoice state | | ||
| | `413` | Request Too Large | Request exceeds maximum element count | | ||
| | `422` | Unprocessable Entity | Invalid payload attributes | | ||
|
|
||
| ### Server errors | ||
|
|
||
| | Status | Meaning | Usage | | ||
| | - | - | - | | ||
| | `500` | Internal Server Error | Unexpected server error | | ||
|
|
||
| Error responses use `Content-Type: text/plain` with a human-readable error message body. |
There was a problem hiding this comment.
The existing Engine API has meaningful machine-readable error codes:
- unknown payload
- invalid forkchoice state
- invalid attributes
- too large request
- unsupported fork
The SSZ HTTP version partly maps these, but not completely:
- unsupported fork is missing
- invalid params vs malformed SSZ are not fully separated
- text/plain error bodies are not machine-stable
Would suggest a normative mapping from JSON-RPC error codes to HTTP status + a small structured error body, even if the success path stays raw SSZ.
|
|
||
| When a new fork introduces a new method version, a new versioned endpoint is added. Older versioned endpoints **MAY** be deprecated but **SHOULD** remain available for backwards compatibility. | ||
|
|
||
| ### Negotiation and fallback |
There was a problem hiding this comment.
Negotiation says “if both advertise, use SSZ”, but what if:
- EL advertises endpoint support but returns 404/415/500
- one endpoint version is implemented incorrectly while others work
Do we:
- permanently downgrade the whole transport?
- downgrade only that method version?
- retry JSON-RPC immediately?
Without clear rules for negotiation and fallback behavior for each client may diverge.
…llable types
- Drop misleading "idempotent" claim on GET /payloads/{payload_id} and
require Cache-Control: no-store; the payload mutates until the slot
deadline so caches/intermediaries must not store or revalidate it.
- Expand security considerations with explicit DoS guidance: pre-read
Content-Length rejection, length/offset validation before allocation,
and operationally enforced per-endpoint body caps. The protocol-level
maxima bound on-chain validity, not per-request resource use.
- Encode truly-nullable fields per the documented List[T, 1] rule:
PayloadStatusV1.latest_valid_hash, ForkchoiceUpdatedResponseV1.payload_id,
and ExecutionPayloadBodyV2.block_access_list. Restores parity with the
JSON spec (each is non-required / oneOf null) and removes the
zero-sentinel ambiguity. Example response length updated 37 -> 41.
| | `Content-Type` (request) | `application/octet-stream` | SSZ-encoded request container | | ||
| | `Content-Type` (response) | `application/octet-stream` | SSZ-encoded response (success) | | ||
| | `Content-Type` (response) | `text/plain` | Human-readable error message | | ||
| | `Accept` (request) | `application/octet-stream` | Client accepts SSZ-encoded responses | |
There was a problem hiding this comment.
I would also like that we support both application/json and application/octet-stream. The application/json would just the existing JSON-RPC with layer with REST interface. That will be very easy for everyone to implement. Later we can add application/octet-stream support for individual endpoint.
|
The motivation for binary SSZ is clear and the performance gains it promises, especially with blobs, are significant and necessary for the protocol's evolution. However, I have some concerns about coupling the transport layer migration (JSON-RPC to REST) with the encoding semantics change (JSON/hex to SSZ). I propose a staged approach that separates these concerns, allowing for a more robust and predictable transition for CL and EL clients. Core Recommendation: Implement REST Transport with Current JSON Semantics First, Then Introduce SSZ Gradually. This approach involves two main phases: Phase 1: Introduce REST Transport with Existing JSON Semantics
Phase 2: Introduce Optional Binary SSZ Encoding Gradually, Endpoint-by-Endpoint
This phased approach embodies the principle of "slow is smooth, smooth is fast" for protocol evolution. It allows implementers to digest one architectural change at a time, leading to a more stable and predictable ecosystem. |
…ConfigurationV1 - Remove INVALID_BLOCK_HASH from the PayloadStatusV1 status enum. The value only applied to engine_newPayloadV1 (Paris) and was supplanted by INVALID starting Shanghai (V2+); all SSZ endpoints in this spec are V2+, so it was dead weight. Addresses @LukaszRozmej review. - Remove the TransitionConfigurationV1 container, the /engine/v1/transition-configuration endpoint section, the endpoint-summary row, and both TOC entries. The corresponding engine_exchangeTransitionConfigurationV1 method was deprecated in Cancun (clients MAY remove support).
|
Alternative: #793 |
I would change transport first and then later we can refactor. Not a fan of doing 2 things at once. This way we can reuse handler code and focus on transport layer. We already implemented this one but we can also support yours if it gains traction |
I will post this here but also copy this message over to the PR:
here is how versioning should be done in REST: https://restfulapi.net/versioning/, we will still need different versions for the containers. getting rid of the versioning just makes everything harder and overall worse for no benefit (you still need different containers) |
Adds src/engine/witness-retrieval.md describing an Engine API extension that allows the consensus layer to opt in to receiving the execution witness alongside the existing payload validation and payload production flows. Design summary: - engine_newPayloadV6: extends V5 with an optional `requestWitness` param and an optional `executionWitness` field on PayloadStatusV2. - engine_forkchoiceUpdatedV5: accepts PayloadAttributesV5, which adds a `requestWitness` field so the EL can configure witness collection before execution begins. - engine_getPayloadV7: response gains an optional `executionWitness`, populated when the originating FCU set `requestWitness = true`. - ExecutionWitnessV1: DATA representing SSZ-encoded witness bytes, following the on-wire convention of BlobV1. Container schema defined externally in a consensus-specs companion document. Offered as an alternative to the *WithWitness method approach (issue ethereum#741, PR ethereum#773). Rationale doc compares this against *WithWitness and notes composition with the SSZ-REST transport (PR ethereum#764).
Adds src/engine/witness-retrieval.md describing an Engine API extension that allows the consensus layer to opt in to receiving the execution witness alongside the existing payload validation and payload production flows. Design summary: - engine_newPayloadV6: extends V5 with an optional `requestWitness` param and an optional `executionWitness` field on PayloadStatusV2. - engine_forkchoiceUpdatedV5: accepts PayloadAttributesV5, which adds a `requestWitness` field so the EL can configure witness collection before execution begins. - engine_getPayloadV7: response gains an optional `executionWitness`, populated when the originating FCU set `requestWitness = true`. - ExecutionWitnessV1: DATA representing SSZ-encoded witness bytes, following the on-wire convention of BlobV1. Container schema defined externally in a consensus-specs companion document. Offered as an alternative to the *WithWitness method approach (issue ethereum#741, PR ethereum#773). Rationale doc compares this against *WithWitness and notes composition with the SSZ-REST transport (PR ethereum#764).
Did you even read my pr? |
Aligns the SSZ schema with PR ethereum#796 (a22fbd4), which added targetGasLimit to PayloadAttributesV4 in the JSON spec but left this SSZ doc behind. Without it, CLs sending the alpha-8 field over SSZ get rejected by EL SSZ decoders that still use the old 6-field layout (off-by-8 withdrawals offset -> malformed SSZ body).
There was a problem hiding this comment.
Following up on my earlier review comment about the capability vocabulary mix: the spec text and reference implementations have moved on, but this specific concern is still unaddressed.
The existing spec defines engine_exchangeCapabilities very explicitly in common.md
Response: Array of string — Array of strings, each string is a name of a method supported by execution layer client software.
Request and response lists MUST contain Engine API methods that are currently supported by consensus and execution client software respectively. Name of each method in both lists MUST include suffixed version.
This PR adds a second grammar <METHOD> /<path> into the same field, so a spec-compliant EL with SSZ support now returns one flat array like:
[
"engine_newPayloadV4",
"engine_newPayloadV5",
"engine_forkchoiceUpdatedV4",
"POST /engine/v5/payloads",
"GET /engine/v6/payloads/{payload_id}",
"POST /engine/v4/forkchoice"
]
That's a literal MUST violation of the existing common.md text — the new entries are not method names and don't carry version suffixes. The PR introduces the new behavior in ssz-encoding.md but leaves the normative definition in common.md unchanged, so the spec becomes internally contradictory if we merge this PR.
Beyond the contradiction, this is a vocabulary change to a protocol-level discovery mechanism, not a transport detail. Existing consumers of engine_exchangeCapabilities (tooling, monitoring, CL-side capability matchers) will see entries that don't fit the established grammar — silently dropping them, failing validation, or misclassifying them depending on implementation.
Notably, the PR already defines a clean SSZ container at POST /engine/v1/capabilities for the same purpose, but it's unreachable until after JSON-RPC negotiation has succeeded — so it doesn't help the bootstrap case the mixing was introduced to solve.
So there are only two choices I see reasonable here:
-
Accept the protocol change explicitly. Update common.md to broaden the response grammar of
engine_exchangeCapabilitiesand retract thename of a method/ version-suffixMUST. Every existing implementation then has to audit its capability consumers for compatibility with the new entry shape. -
Add a new dedicated method Add
engine_exchangeSszEndpointsV1, returning SSZ-REST endpoint identifiers only. EL support for SSZ-REST is advertised through the existingengine_exchangeCapabilitiesby includingengine_exchangeSszEndpointsV1in the response. The CL then calls that method to fetch REST endpoint support through a clean surface. Legacy consumers see only another normal Engine API method name with a version suffix, so the existingengine_exchangeCapabilitiescontract remains intact.
## Summary Implements SSZ-REST Engine API transport on the consensus layer (client side), as specified in [ethereum/execution-apis#764](ethereum/execution-apis#764). - New CLI flag `--execution.sszRestUrl` to configure SSZ-REST endpoint - SSZ-encoded request/response bodies for all Engine API methods - Automatic fallback to JSON-RPC on network errors - Supports: `new_payload` (v1-v5), `forkchoice_updated` (v1-v3), `get_payload` (v1-v5), `exchange_capabilities` - Proper fork-based version selection for Deneb/Electra/Fulu --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary Implements SSZ-REST Engine API transport on the consensus layer (client side), as specified in [ethereum/execution-apis#764](ethereum/execution-apis#764). - New CLI flag `--execution.sszRestUrl` to configure SSZ-REST endpoint - SSZ-encoded request/response bodies for all Engine API methods - Automatic fallback to JSON-RPC on network errors - Supports: `new_payload` (v1-v5), `forkchoice_updated` (v1-v3), `get_payload` (v1-v5), `exchange_capabilities` - Proper fork-based version selection for Deneb/Electra/Fulu --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…sszRest The SSZ-REST Engine API transport from #8994 was constructed unconditionally and probed on every Engine call, then silently fell back to JSON-RPC on network errors. Until ethereum/execution-apis#764 stabilises and the ELs we test against advertise support consistently, this probing is wasted traffic against vanilla EL deployments and can mask transient infra issues. Add a `sszRest` flag to ExecutionEngineHttpOpts and a hidden `--execution.sszRest` CLI flag. The SszRestClient is only constructed when the flag is set; otherwise the JSON-RPC path is used exclusively. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary Implements SSZ-REST Engine API transport on the consensus layer (client side), as specified in [ethereum/execution-apis#764](ethereum/execution-apis#764). - New CLI flag `--execution.sszRestUrl` to configure SSZ-REST endpoint - SSZ-encoded request/response bodies for all Engine API methods - Automatic fallback to JSON-RPC on network errors - Supports: `new_payload` (v1-v5), `forkchoice_updated` (v1-v3), `get_payload` (v1-v5), `exchange_capabilities` - Proper fork-based version selection for Deneb/Electra/Fulu --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…sszRest The SSZ-REST Engine API transport from #8994 was constructed unconditionally and probed on every Engine call, then silently fell back to JSON-RPC on network errors. Until ethereum/execution-apis#764 stabilises and the ELs we test against advertise support consistently, this probing is wasted traffic against vanilla EL deployments and can mask transient infra issues. Add a `sszRest` flag to ExecutionEngineHttpOpts and a hidden `--execution.sszRest` CLI flag. The SszRestClient is only constructed when the flag is set; otherwise the JSON-RPC path is used exclusively. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Core change: Full binary SSZ over REST. No JSON, no hex encoding - raw SSZ bytes over HTTP.
Rationale: